Deep Neural Network Approximation Theory

This paper develops fundamental limits of deep neural network learning by characterizing what is possible if no constraints are imposed on the algorithm and amount training data. Concretely, we consider Kolmogorov-optimal approximation through networks with guiding theme being a relation between complexity function (class) to be approximated approximating in terms connectivity memory requirements for storing topology associated quantized weights. The theory develop establishes that approximants markedly different classes, such as unit balls Besov spaces modulation spaces. In addition, provide exponential accuracy-i.e., error decays exponentially number nonzero weights network-of multiplication operation, polynomials, sinusoidal functions, certain smooth functions. Moreover, this holds true even one-dimensional oscillatory textures Weierstrass function-a fractal function, neither which has previously known methods achieving accuracy. We also show sufficiently functions finite-width require strictly smaller than finite-depth wide networks.